What Works Clearinghouse™ 


U.S. DEPARTMENT OF EDUCATION 


111 WWC Intervention Report /joe 

a summary of findings from a systematic review of the evidence 


Secondary Mathematics 


systematic 


INSTITUTE of 

EDUCATION SCIENCES 


August 2017* 


I CAN Learn® 


Intervention Description 1 

I CAN Learn ® is a computer-based math curriculum for students in 
middle school, high school, and college. It provides math instruction 
through a series of interactive lessons that students work on individually 
at their own computers. Students move at their own pace and must 
demonstrate mastery of each concept before progressing to the next 
one. Classroom teachers may provide individual, small-group, or whole- 
class instruction based on students’ performance on the software 
program. This review focuses on studies of / CAN Learn®’s secondary 
courses, / CAN Learn ® Algebra and / CAN Learn ® Geometry. 

Research 2 

The What Works Clearinghouse (WWC) identified one study of / CAN 
Learn® Algebra that both falls within the scope of the Secondary 
Mathematics topic area and meets WWC group design standards. No 
studies of / CAN Learn® Algebra meet WWC group design standards 
without reservations, and one study of / CAN Learn® Algebra meets 
WWC group design standards with reservations. This study included 
686 students in eighth grade in one school district. 

The WWC considers the extent of evidence for / CAN Learn® Algebra on 
the mathematics test scores of secondary students to be small for one 
outcome domain—general mathematics achievement. No studies meet WWC group design standards in the five other 
domains, so this intervention report does not report on the effectiveness of / CAN Learn® for those domains. 3 (See the 
Effectiveness Summary on p. 5 for more details of effectiveness by domain. 

The findings in this report pertain only to / CAN Learn® Algebra. No studies that examine / CAN Learn® Geometry fall 
within the scope of the Secondary Mathematics review protocol and meet WWC group design standards. 

Effectiveness 

I CAN Learn® Algebra had no discernible effects on general mathematics achievement for secondary students. 
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Table 1. Summary of findings 4 




Improvement index 

(percentile points) 




Course and outcome 
domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

1 CAN Learn® Algebra 

General mathematics achievement 

No discernible 
effects 

+5 

na 

1 

686 

Small 

1 CAN Learn ® Geometry 

No evidence 







na = not applicable. 
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Intervention Information 

Background 

JRL Enterprises, Inc. is the developer and distributor of / CAN Learn®. Address: 912 Constantinople Street, New 
Orleans, LA 70115. Email: info@icanlearn.com. Website: http://www.icanlearn.com. Telephone: (504) 263-1380. 

Intervention details 

I CAN Learn® is a full-curriculum mathematics software program. It is available in online learning models and 
in traditional classroom models. I CAN Learn® provides math instruction through a series of computer-based 
interactive lessons that students work on individually at their own computers in middle school, high school, and 
college classrooms. Students typically start with a placement test, which is used to assess their current level of 
knowledge and customize their lesson plan. Each lesson includes a warm-up activity, lesson presentation, guided 
practice, and a quiz to ensure mastery of the lesson content. A virtual teacher in the software presents the lessons 
and demonstrates how to solve problems. Students then work at their own pace and must demonstrate mastery 
of each concept before progressing to the next one. Classroom teachers may provide individual, small-group, or 
whole-class instruction based on students’ performance on the software program. Printed textbooks are available 
to supplement the electronic courseware. 

I CAN Learn® courses include Fundamentals of Math, Pre-Algebra, Algebra, and Geometry. I CAN Learn® Algebra 
and Geometry are examined as part of this secondary mathematics review. According to the developer, all four 
courses align to the National Council of Teachers of Mathematics standards and can be customized to meet state- 
or district-specific standards. Each course contains 20 to 180 self-paced, mastery-based lessons. Lessons include 
multimedia content, such as instructional videos and animation. In addition to the instructional content, / CAN 
Learn® enables teachers to conduct classroom administration tasks through the / CAN Learn® Classroom Explorer 
Class Management System, which tracks students’ attendance, homework, and test grades, and can create an 
individualized learning path for each student. In addition, teachers can use the system to generate a variety of 
reports that support instructional decisions, such as identifying areas in which students need assistance; grouping 
students; and tracking attendance, homework, and grades. 


Cost 

The cost of / CAN Learn® depends on its configuration and terms of support. As of March 2017, using a school’s 
existing hardware, subscriptions provide access to more than 600 lessons and cost from $10 per student for site 
licenses to $53.48 for an individual student license. Varying support plans, including training, professional develop¬ 
ment, curriculum alignments, implementation planning, and other pedagogical support, are available and cost from 
$400 to $20,000 per year. More detailed information about cost is available from the developer. 
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Research Summary 

The WWC identified 12 studies of / CAN Learn ® secondary courses Table 2. Scope of reviewed research 
that were eligible for review: 

• All 12 eligible group design studies investigated the effects 
of / CAN Learn® Algebra on the mathematics test scores of 
secondary students. 

• No eligible group design studies investigated the effects of / CAN Learn® Geometry on the mathematics test 
scores of secondary students. 5 

The WWC reviewed all 12 eligible studies against group design standards. One study uses a quasi-experimental 
design that meets WWC group design standards with reservations. This report summarizes the one study that 
focuses on / CAN Learn® Algebra. The remaining 11 studies of / CAN Learn® Algebra do not meet WWC group 
design standards. 

An additional 28 studies of / CAN Learn® were identified but do not meet WWC eligibility criteria (see the Glossary 
of Terms in this document for a definition of this term and other commonly used research terms) for review in this 
topic area. Citations for all 40 studies are in the References section, which begins on p. 6. 

Summary of I CAN Leanfi Algebra studies meeting WWC group design standards without reservations 

No studies of / CAN Learn® Algebra meet WWC group design standards without reservations. 

Summary of I CAN Leanfi Algebra study meeting WWC group design standards with reservations 

Kerstyn (2001) used a quasi-experimental design to examine the effects of / CAN Learn® Algebra on eighth-grade 
students in 11 middle schools in one school district in the 2000-01 school year. Five teachers in the study schools 
taught / CAN Learn® Algebra to eight Algebra I classes, and four teachers in the study schools taught / CAN Learn® 
Algebra to eight Algebra I Honors classes. District staff matched the 16 / CAN Learn® classrooms to 16 similar 
classrooms that did not use / CAN Learn®. These comparison classrooms included six teachers who taught eight 
Algebra I classes and four teachers who taught eight Algebra I Honors classes. The study included 350 Algebra I 
students and 336 Algebra I Honors students. The study used the Florida Comprehensive Assessment Test math 
assessment to measure student achievement outcomes for eighth-grade algebra. The study did not specify the 
edition of / CAN Learn® Algebra used. 


Grade 

8 

Delivery method 

Whole class 

Program type 

Curriculum 
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Effectiveness Summary 

The WWC review of / CAN Learn ® Algebra for the Secondary Mathematics topic area includes student outcomes 
in six domains: algebra, geometry, statistics and probability, trigonometry/precalculus, calculus, and general 
mathematics achievement. The one study of / CAN Learn® Algebra that meets WWC group design standards 
reported findings in one of the six domains: general mathematics achievement. The following findings present 
the author’s estimates and WWC-calculated estimates of the size and statistical significance of the effects of / 

CAN Learn® Algebra on eighth-grade student test scores. Additional comparisons are presented as supplemental 
findings for / CAN Learn® Algebra in Appendix D. These supplemental findings do not factor into / CAN Learn® 
Algebra’s rating of effectiveness. For a more detailed description of the rating of effectiveness and extent of 
evidence criteria, see the WWC Rating Criteria on p. 18. 

Summary of / CAN Learn® Algebra effectiveness for the general mathematics achievement domain 

Table 3. Rating of effectiveness of I CAN Learn® Algebra and extent of evidence for the general mathematics 
achievement domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

No affirmative evidence of effects. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the general 
mathematics achievement domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 686 students in 11 schools reported evidence of effectiveness in the general mathematics 
achievement domain. 


One study of / CAN Learn® Algebra that meets WWC group design standards with reservations reported findings in 
the general mathematics achievement domain. 

Kerstyn (2001) reported, and the WWC confirmed, no statistically significant difference between the / CAN Learn® 
Algebra group and the comparison group in the general mathematics achievement domain. The effect size was not 
large enough to be considered substantively important according to WWC criteria (that is, an effect size of at least 
0.25). The WWC characterizes this study finding as an indeterminate effect. 

Thus, for the general mathematics achievement domain, one study of / CAN Learn® Algebra showed an 
indeterminate effect. This results in a rating of no discernible effects, with a small extent of evidence. 
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Appendix A: Research details for Kerstyn (2001) 

Kerstyn, C. (2001). Evaluation of the I CAN Learn mathematics classroom: Second year of implementation 

(2000-2001 school year). Tampa, FL: Hillsborough County Public Schools. 

Additional sources: 

Kerstyn, C. (2002). Evaluation of the I CAN Learn mathematics classroom: Second year of 

implementation (2001-2002 school year). Tampa, FL: Hillsborough County Public Schools. 

Kerstyn, C. (2004). Teachers’ mathematics preparation and eighth grade student mathematics 

achievement: Can an integrated learning system provide support when teachers’ professional 
preparation is limited? (Doctoral dissertation, University of South Florida). 

Table A. Summary of / CAN Leam® Algebra findings Meets WWC group design standards 

with reservations 


Study findings 

Outcome domain 

Sample size 

Average improvement index 

(percentile points) Statistically significant 

General mathematics achievement 

32 classrooms/ 

686 students 

+5 No 


Setting The study was conducted in 16 eighth-grade classrooms across 11 middle schools in Hillsborough 
County Public School District in Florida. The study occurred in the 2000-01 school year. 

Study sample The district implemented / CAN Learn ® Algebra and / CAN Leam® Pre-Algebra in select 
classrooms within 12 middle schools during the 2000-01 school year. For the evaluation, 
within each study school, district staff selected comparison classrooms that used traditional 
instruction and matched them on several factors including: students’ prior achievement, time 
of day, instructional time, class size, and proportion of minority students. This review is based 
on the / CAN Learn® Algebra classrooms in 11 of those schools, which included nine / CAN 
Learn® Algebra teachers (five for Algebra I and four for Algebra I Honors) and 10 comparison 
teachers (six for Algebra I and four for Algebra I Honors). 6 


There were 686 students in the study: 350 Algebra I students and 336 Algebra I Honors students. 
All of the Algebra I and Algebra I Honors students were general education students. Across all 
algebra students in the study, approximately 25% qualified for free or reduced-price lunch, 52% 
are female, approximately 53% were White, 20% were Black, and 19% were Hispanic. 

Intervention Intervention students were taught using / CAN Learn® Algebra as the primary source of 

group instruction for the entire academic school year. The curriculum includes 109 algebra lessons, 

each of which has a five-part format that includes a warm-up activity, lesson presentation, 
journal activity, guided practice, and a quiz to ensure mastery of the lesson content. Students 
complete the lessons individually and at their own pace using interactive software with a 
virtual teacher that presents the multimodal lessons and demonstrates how to solve a problem 
if students make errors. In each class, there is a classroom teacher who supported students 
with the lessons. The study did not specify which edition of the curriculum was used. 
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Comparison 

group 

Outcomes and 
measurement 


Support for 
implementation 


Comparison students used a traditional math curriculum already in place in the district. The 
study did not describe or name the comparison curriculum. 

The primary outcome measure is the 2001 Florida Comprehensive Assessment Test (FCAT) 
mathematics exam aligned with Florida’s Sunshine State Standards (SSS). The outcome is 
measured using the eighth grade scale score. The seventh grade FCAT Norm Reference Test- 
Normal Curve Equivalent (NRT-NCE) scores from 2000 were used as the baseline assessment. 

The study also presented outcomes on the district’s end-of-semester algebra exam in the 
2000-01 school year, which students take at the end of the first semester. Because this 
assessment measures student achievement mid-year, it is considered a supplemental 
outcome that does not factor into the intervention’s rating of effectiveness. For a more detailed 
description of both outcome measures, see Appendix B. 

The study presents several outcomes that are ineligible for review under the Secondary 
Mathematics review protocol, including teacher attitudes, parent attitudes, and student 
attitudes. In addition, the study presents first semester tests and FCAT scores for prealgebra 
students; these are ineligible for review, since prealgebra falls under the Primary Mathematics 
topic area. 

The study did not specify how much training intervention teachers received. The district 
contracted with JRL Enterprises, Inc. to use / CAN Learn® during the year prior to the study 
(1999-2000). To implement the curriculum, classrooms were equipped with desks, computer 
equipment, and electrical connectivity. JRL Enterprises, Inc. provided maintenance on the 
equipment and technical and instructional support to teachers. 
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Appendix B: Outcome measures for each domain 


Algebra 

End-of-Semester Algebra Exam 

The end-of-semester algebra exam was developed by the Hillsborough County School District in Florida, All 
algebra students in the district are required to take the assessment, and it counts as a third of their semester 
grade (as cited in Kerstyn, 2001). In response to a WWC query, the author indicated the exam has an internal 
consistency between .84 to .88. This outcome is reported as a supplemental finding in Appendix D. 

General mathematics achievement 

Florida Comprehensive Assessment Test 
(FCAT) Mathematics exam 

The FCAT Mathematics exam is a standardized assessment that includes items aligned to all five content 
strands in the Florida Sunshine State Standards: (1) number sense, concepts, and operations; (2) measurement; 
(3) geometry and spatial sense; (4) algebraic thinking; and (5) data analysis and probability. Prior to 2001, 
the FCAT math score was a composite score that combined student performance on the multiple choice and 
performance portions of the math test. The 2001 FCAT scale score is based only on the multiple choice portion 
of the test. The assessment administered in the 2001-02 school year had a reliability ranging from .89 to .93. 
The author reports the correlation between the 2000 NCE score and 2001 Scale score as 0.512 (as cited in 
Kerstyn, 2001,2002, and 2004). 
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Appendix C: Findings included in the rating for studies of I CAN Learif Algebra for the mathematics 
achievement domain 





Mean 

(standard deviation) 

WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Kerstyn (2001 ) a 

2001 FCAT Math 
Scale Score 

Grade 8 
Algebra 1 & 
Algebra 1 Honors 

32 classrooms/ 
686 students 

363.81 

(33.76) 

359.25 

(34.32) 

4.56 

0.13 

+5 

.71 

Domain average for mathematics achievement (Kerstyn, 2001) 



0.13 

+5 

Not 

statistically 

significant 

Domain average for mathematics achievement across all studies 



0.13 

+5 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding. FCAT = Florida Comprehensive Assessment Test, na = not applicable. 

a For Kerstyn (2001), the author reported unadjusted FCAT means and standard deviations separately by Algebra I and Algebra I Honors classes. The WWC pooled these classes together to 
estimate an overall effect size. The WWC did not need to make corrections for clustering or multiple comparisons. The WWC calculated the intervention group mean using a difference-in- 
differences approach by adding the impact of the intervention (i.e., difference in mean gains between the intervention and comparison groups) to the unadjusted comparison group posttest 
means. The p-value presented here was calculated by the WWC. The author used classroom-level means to estimate effects and made inferences about the effect of / CAN Learn 0 Algebra 
on classroom achievement. The WWC used the student-level posttest standard deviations to estimate an effect size. The intervention and comparison group means in this table differ 
from the prior Middle School Math intervention report because this table pooled together the Algebra I and Algebra I Honors classes, which was not done by the prior review. This study is 
characterized as having an indeterminate effect because the estimated effect is neither statistically significant nor substantively important (0.25 standard deviations or larger). For more 
information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix D: Description of supplemental findings of / CAN Learn® Algebra for the mathematics 
achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Kerstyn, 2001 a 

2001 FCAT Math 

Scale Score 

Grade 8: 
Algebra 1 

16 classrooms/ 
350 students 

351.98 

(30.80) 

345.40 

(28.36) 

6.58 

0.21 

+8 

.19 

2001 FCAT Math 

Scale Score 

Grade 8: 
Algebra 1 
Honors 

16 classrooms/ 
336 students 

nr 

nr 

na 

na 

na 

.69 

2001 End-of- 
Semester Algebra 

Exam 

Grade 8: 
Algebra 1 

16 classrooms/ 
350 students 

nr 

nr 

na 

na 

na 

.68 

2001 End-of- 
Semester Algebra 

Exam 

Grade 8: 
Algebra 1 
Honors 

16 classrooms/ 
336 students 

nr 

nr 

na 

na 

na 

.69 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. FCAT = Florida Comprehensive Assessment Test, na = not applicable, nr = not reported 

a For Kerstyn (2001), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The WWC calculated the 
intervention group mean using a difference-in-differences approach by adding the impact of the intervention (i.e., difference in mean gains between the intervention and comparison 
groups) to the unadjusted comparison group posttest means. The p-values presented here were reported in the original study. The intervention and comparison group means in this table 
differ from the prior Middle School Math intervention report because this table presents class-level means and student-level standard deviations. The prior intervention report, based on the 
version 1.0 Standards, used student-level means. The current standards (version 3.0) clarified that baseline equivalence must be assessed on the analytic sample. The study reported only 
classroom-level baseline data, so only the analysis of classroom-level means meets WWC group design standards. For the other three analytic samples in the table (that is, the samples 
used in the analysis), the author reported p-values using the results from an ANCOVA model, but did not report the information needed to calculate a WWC effect size. 
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Endnotes 

' Due to the 2015 restructuring of the Mathematics topic area from three areas (Elementary, Middle, and High School) to two areas 
(Primary and Secondary Mathematics), this is considered a new report, rather than an updated report. The information in this report 
combines the research examined in the prior reports and presents the conclusions differently. 

1 The descriptive information for this intervention comes from a publicly available source—the developer’s website (www.icanlearn.com, 
downloaded November 2016). The What Works Clearinghouse (WWC) provided the developer with the intervention description in 
December 2016, and asked the developer to review it for accuracy from their perspective. The WWC subsequently incorporated 
feedback from the developer. Further verification of the accuracy of the descriptive information for this intervention is beyond the 
scope of this review. 

2 The WWC previously released reports on / CAN Learn ® under the Middle School Mathematics (MSM) topic area in March 2009 and 
the High School Mathematics (HSM) topic area in February 2012; the WWC prepared the reports using the WWC Procedures and 
Standards Handbook (versions 1.0 and 2.0, respectively), and the Middle and High School Mathematics review protocols (versions 

1.0 and 2.1, respectively). In June 2015, the WWC restructured the reviews of research on math interventions into two areas instead 
of three. These two review areas are Primary Mathematics (which includes interventions in which math is presented through multi¬ 
topic materials and curricula, typically used in grades K—8), and Secondary Mathematics (which includes interventions organized by 
math content area [e.g., algebra, geometry, and calculus], typically taught in grades 9-12). These two areas are replacing the prior 
Elementary School Mathematics, MSM, and HSM topic areas, which were organized by student grade level. The WWC is updating 
and replacing intervention reports written under the prior topic areas. 

The literature search reflects documents publicly available by November 2016. This updated report includes reviews of 14 studies that the 
previous WWC intervention reports did not include. Of the additional studies, 12 were not within the scope of the review protocol for the 
Secondary Mathematics topic area, and two were within the scope of the review protocol for the Secondary Mathematics topic area but 
did not meet WWC group design standards. A complete list and disposition of all studies reviewed are available in the references. 

The current report includes reviews of all previous studies that met WWC group design standards with or without reservations, and 
resulted in a revised disposition for 6 studies. 

Barrow et al. (2009) is rated does not meet WWC group design standards in this report, whereas it had previously received a rating 
of meets WWC group design standards with reservations in the HSM intervention report and a rating of does not meet WWC group 
design standards in the MSM intervention report. The prior reviews focused on results by middle and high school (grade level), 
and were conducted based on versions 1.0 and 2.1 of the WWC Procedures and Standards Handbook for the MSM and HSM 
reports, respectively. In both prior reviews, the WWC found the study to have high attrition; therefore, the study had to demonstrate 
equivalence. In response to a query for the HSM review, the authors provided evidence of equivalence for the analytic sample (that is, 
the sample used for study analysis) reviewed for the HSM report. The authors did not respond to a similar WWC query for the MSM 
report. Therefore, the two prior ratings differed. In the current review based on the revised Secondary Mathematics topic area review 
protocol, the review focused on students in algebra courses, regardless of grade level. The WWC sent a question to the authors for 
attrition information and evidence of baseline equivalence for the eligible sample of students in algebra courses, but the authors did 
not provide this information; therefore, attrition could not be assessed and baseline equivalence was not demonstrated. As a result, 
the study does not meet standards. 

Parrott (2005) is ineligible for review in the current report, whereas it was rated does not meet WWC group design standards in the 
previous MSM intervention report based on version 1.0 of the WWC Procedures and Standards Handbook. The rating has changed 
due to a change in WWC eligibility rules under version 3.0 of the Standards. Under the version 3.0 Standards, master’s theses are not 
eligible for review; therefore, the study is no longer eligible for review. 

The remaining six studies with rating changes are ineligible for review in this report, whereas they were rated does not meet WWC 
group design standards in a previous report. In all six cases, the change in rating was due to the restructuring of the Mathematics 
topic area from three areas (Elementary, Middle, and High) to two areas (Primary and Secondary). The prior topic areas were defined 
by grade level, whereas the current topic areas are defined by course content. 

(1) Kirby (2003) is ineligible for review in this report, whereas it was rated does not meet WWC group design standards in the 
previous MSM report. The prior review was based on version 1.0 of the WWC Procedures and Standards Handbook. The 
study includes eighth-grade students, and the author does not indicate whether they used / CAN Learn ® Algebra or / CAN 
Learn® Pre-Algebra. The WWC questioned the author to clarify the intervention used but did not receive a response. Because 
it is unknown whether the study uses an eligible intervention, the study is ineligible for review. 

(2) Kirby (2004d) is ineligible for review in this report, whereas it was previously rated meets WWC group design standards with 
reservations in the MSM intervention report based on version 1.0 of the WWC Procedures and Standards Handbook. In the 
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current report, the study is ineligible for review because the intervention examines the effectiveness of / CAN Learn ® Pre- 
Algebra, which is not eligible for review under the Secondary Mathematics topic area. 

(3) Kirby (2004e) is ineligible for review in this report, whereas it was rated meets WWC group design standards with reservations 
in the previous MSM report. The prior review was based on version 1.0 of the WWC Procedures and Standards Handbook. 
The study includes eighth-grade students, and the author does not indicate whether / CAN Learn 9 Algebra or / CAN Learn 9 
Pre-Algebra was used. The WWC questioned the author to clarify the intervention used but did not receive a response. 
Because it is unknown whether the study uses an eligible intervention, the study is ineligible for review. 

(4) Kirby (2006d) is ineligible for review in this report, whereas it was rated meets WWC group design standards with reservations 
in the previous MSM report. The prior review was based on version 1.0 of the WWC Procedures and Standards Handbook. 
The study includes eighth-grade students, and the author does not indicate whether / CAN Learn 9 Algebra or / CAN Learn 9 
Pre-Algebra was used. The WWC questioned the author to clarify the intervention used but did not receive a response. 
Because it is unknown whether the study uses an eligible intervention, the study is ineligible for review. 

(5) Oescher (2002) is ineligible for review in this report, whereas it was rated does not meet WWC group design standards in 
the previous MSM and HSM reports. The prior reviews were based on versions 1.0 and 2.1 of the WWC Procedures and 
Standards Handbook, respectively. The study includes eighth-grade students, and the author does not indicate whether 
I CAN Learn 9 Algebra or / CAN Learn ® Pre-Algebra was used. The WWC questioned the author to clarify the intervention 
used but did not receive a response. Because it is unknown whether the study uses an eligible intervention, the study is 
ineligible for review. In addition, Oescher (2002) is considered a related citation for JRL Enterprises, Inc. (n.d.b) in this report. 

(6) Scafide (2004) is ineligible for review in this report, whereas the study was rated does not meet WWC group design standards 
in the previous MSM intervention report. The prior review was based on version 1.0 of the WWC Procedures and Standards 
Handbook. The study includes eighth-grade students, and the author does not indicate whether / CAN Learn 9 Algebra or 

I CAN Learn ® Pre-Algebra was used. The WWC questioned the author to clarify the intervention used but did not receive 
a response. Because it is unknown whether the study uses an eligible intervention, the study is ineligible for review. 

In addition to the rating changes, this review effort identified some citations to be related and therefore to comprise a single study, 
whereas the prior reports treated the citations as separate studies. These changes are due to clarification of a study definition in 
version 3.0 of the WWC Standards, and are notable in terms of the effectiveness rating and WWC-reported findings for Kerstyn (2001), 
Kerstyn (2002), and Kerstyn (2004), which are considered related citations for one study in this report. In the prior MSM report, the 
WWC treated Kerstyn (2001) and Kerstyn (2002) as separate studies, and Kerstyn (2004) was not included. These studies are now 
considered multiple citations from one study, and they collectively received the rating of meets WWC group design standards with 
reservations, which is the same rating in the current and prior reports. 

Reviews of studies in this report used the standards from the WWC Procedures and Standards Handbook (version 3.0) and the 
Primary Mathematics topic area review protocol (version 3.1). The evidence presented in this report is based on available research. 
Findings and conclusions may change as new research becomes available. 

3 Please see the Secondary Mathematics review protocol (version 3.1) for a list of all outcome domains. 

4 For criteria used to determine the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 18. These 
improvement index numbers show the average and range of individual-level improvement indices for the finding in the study. 

5 No studies examining the effectiveness of / CAN Learn 9 Geometry fall within the scope of the Secondary Mathematics review 
protocol and meet WWC group design standards. Because no studies meet WWC group design standards at this time, the WWC 
is unable to draw any conclusions based on research about the effectiveness or ineffectiveness of / CAN Learn 9 Geometry on 
secondary students. Additional research that meets WWC standards is needed to determine the effectiveness or ineffectiveness of 
this intervention. 

6 The study continued for a second school year in 2001-02. Kerstyn (2002) presented analytic samples of Algebra I and Algebra I 
Honors classrooms based on the Spring 2002 FCAT outcomes. These 2002 analytic samples are rated does not meet WWC group 
design standards because they do not demonstrate baseline equivalence as required. Kerstyn (2001) and Kerstyn (2002) also 
presented results based on pre-algebra classrooms that were ineligible for review in the Secondary Mathematics topic area. The 
2001-02 data were further analyzed in Kerstyn (2004), which presented results based on pre-algebra students that were ineligible 
for review in the Secondary Mathematics topic area. The results based on pre-algebra students are eligible for review in the Primary 
Mathematics topic area. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 

The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 

The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 


Attrition Attrition occurs when an outcome variable is not available for all subjects initially assigned to 
the intervention and comparison groups. If a randomized controlled trial (RCT) or regression 
discontinuity design (RDD) study has high levels of attrition, the validity of the study results 
can be called into question. An RCT with high attrition cannot receive the highest rating of 
Meets WWC Group Design Standards without Reservations, but can receive a rating of Meets 
WWC Group Design Standards with Reservations if it establishes baseline equivalence of the 
analytic sample. Similarly, the highest rating an RDD with high attrition can receive is Meets 
WWC RDD Standards with Reservations. 

For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and individuals 
leave the group. If a single-case design does not meet minimum requirements for phases and 
data points within phases, the study cannot receive the highest rating of Meets WWC Pilot 
Single-Case Design Standards without Reservations. 

Baseline a point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at 
baseline. In a single-case design experiment, the baseline condition is a period during which 
participants are not receiving the intervention. 

Clustering adjustment An adjustment to the statistical significance of a finding when the units of assignment 

and analysis differ. When random assignment is carried out at the cluster level, outcomes 
for individual units within the same clusters may be correlated. When the analysis is con¬ 
ducted at the individual level rather than the cluster level, there is a mismatch between 
the unit of assignment and the unit of analysis, and this correlation must be accounted for 
when assessing the statistical significance of an impact estimate. If the correlation is not 
accounted for in a mismatched analysis, the study may be too likely to report statistically 
significant findings. To fairly assess an intervention’s effects, in cases where study authors 
have not corrected for the clustering, the WWC applies an adjustment for clustering when 
reporting statistical significance. 


Confounding factor 

Design 


Effect size 
Eligibility 
Equivalence 


A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence of 
an intervention (single-case design). Designs eligible for WWC review are randomized controlled 
trials, quasi-experimental designs, regression discontinuity designs, and single-case designs. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Glossary of Terms 


Extent of evidence An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence cat¬ 
egories: small and medium to large. 

• small: includes only one study, or one school, or findings based on a total sample 
size of less than 350 students and 14 classrooms (assuming 25 students in a class) 

• medium to large: includes more than one study, more than one school, and 
findings based on a total sample of at least 350 students or 14 classrooms 

Gain scores The result of subtracting the pretest from the posttest for each individual in the sample. 

Some studies analyze gain scores instead of the unadjusted outcome measure as a method 
of accounting for the baseline measure when estimating the effect of an intervention. The 
WWC reviews and reports findings from analyses of gain scores, but gain scores do not 
satisfy the WWC’s requirement for a statistical adjustment under the baseline equivalence 
requirement. This means that a study that must satisfy the baseline equivalence requirement 
and has baseline differences between 0.05 and 0.25 standard deviations Does Not Meet 
l/l/l/l/C Group Design Standards if the study’s only adjustment for the baseline measure was 
in the construction of the gain score. 


Group design a study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 


Improvement index Along a percentile distribution of individuals, the improvement index represents the gain or 
loss of the average individual due to the intervention. As the average individual starts at the 
50th percentile, the measure ranges from -50 to +50. 


Intervention An educational program, product, practice, or policy aimed at improving student outcomes. 


Intervention report a summary of the findings of the highest-quality research on a given program, product, 

practice, or policy in education. The WWC searches for all research studies on an intervention, 
reviews each against design standards, and summarizes the findings of those that meet 
WWC design standards. 


Multiple comparison An adjustment to the statistical significance of results to account for multiple comparisons 
adjustment ' n a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 
the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 
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Outcome domain 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Regression discontinuity 
design (RDD) 

Single-case design 
Standard deviation 

Statistical significance 

Study rating 


Substantively important 
Systematic review 


A group of closely-related outcomes. A domain is the organizing construct for a set of related 
outcomes through which studies claim effectiveness. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

For group design research, the WWC rates the effectiveness of an intervention in each domain 
based on the quality of the research design and the magnitude, statistical significance, and 
consistency in findings. For single-case design research, the WWC rates the effectiveness 
of an intervention in each domain based on the quality of the research design and the 
consistency of demonstrated effects. The criteria for the ratings of effectiveness are given 
in the WWC Rating Criteria on p. 18. 

A design in which groups are created using a continuous scoring rule. For example, stu¬ 
dents may be assigned to a summer school program if they score below a preset point on a 
standardized test, or schools may be awarded a grant based on their score on an applica¬ 
tion. A regression line or curve is estimated for the intervention group and similarly for the 
comparison group, and an effect occurs if there is a discontinuity in the two regression lines 
at the cutoff. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statisti¬ 
cally significant if the likelihood that the difference is due to chance is less than 5% (p <.05). 

The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets l/l/l/l /C Design Standards without Reservations, Meets l/l/WC Design Standards with 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 

A review of existing literature on a topic that is identified and reviewed using explicit methods. 
A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their findings; 
4) combining findings within and across studies; and 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention 

Report 



Practice 

Guide 



Quick 

Review 


Single Study 
Review 



An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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